1. Field of the Invention
The invention relates to systems which perform image processing, and more specifically to systems which perform two-dimensional image enhancement algorithms.
2. Discussion of Related Art
Various known teachings which are believed to be related to various ones of the innovations disclosed in the present application will now be discussed. However, applicants specifically note that not every idea discussed in this section is necessarily prior art. For example, the characterizations of the particular patents and publications discussed may relate them to inventive concepts in a way which is itself based on knowledge of some of the inventive concepts. Moreover, the following discussion attempts to fairly present various suggested technical alternatives (to the best of applicants' knowledge), even though the teachings of some of those technical alternatives may not be "prior art" under the patent laws of the United States or of other countries. Similarly, the Summary of the Invention section of the present application may contain some discussion of prior art teachings, interspersed with discussion of generally applicable innovative teachings and/or specific discussion of the best mode as presently contemplated, and applicants specifically note that statements made in the Summary section do not necessarily delimit the various inventions claimed in the present application or in related applications.
A video signal may consist of a series of two-dimensional images ("frames") which are transmitted in rapid succession. A human observer seeing the rapid succession of frames will get the impression of a continuous picture which may include motion.
Each frame may be described as a grid of minimum-dimension picture elements (or "pixels") of varying intensities. That is, the dimensions of the pixels are defined to be at least as small as the maximum resolution of the image, so that the image may be represented by transmitting very simple set of scalar parameters (e.g. grey-scale and color information) for each pixel.
The video signal is normally transmitted by encoding pixel parameters sequentially on an electronic waveform. In the most common versions of this, the pixels will be transmitted in a pixel-by-pixel, serial fashion which corresponds to transmitting one horizontal line of pixels (e.g. from left to right) at a time. The horizontal lines are also sequentially transmitted (e.g., from top to bottom) in groups which correspond to fields. This technique is commonly called raster scanning.
The serial waveform thus transmitted can be converted into a two-dimensional picture by a monitoring device such as a television picture tube (cathode ray tube), wherein the serial sequencing used to transmit the pixels is reproduced as a sequence of setting pixel brightness values (e.g. by steering an electron beam across a display phosphor).
Interlaced scanning is a very common variation of the raster scanning technique. Interlaced scanning involves the "interlacing" of the horizontal lines. Alternate lines are transmitted as a complete field, and the next field contains the lines not transmitted in the previous field. In other words, the first field contains the odd numbered lines and the second field contains the even numbered lines. Thus, two consecutive fields comprise a complete frame. This technique is used to reduce the transmission bandwidth of the video signal.
There are many different configurations and variations of raster scanning. The NTSC standard currently in use in the United States provides 525 horizontal lines per frame and a maximum of 525 pixels per horizontal line, and transmits each horizontal line in 63.5 microseconds. Other standards are currently in use throughout the world. These other standards, such as CCIR, PAL, or SECAM, may have different transmission times, maximum number of pixels per line, and number of lines per frame. Many factors (in addition to compatibility with available signal sources) can be considered when choosing a transmission standard, such as the desired resolution or sharpness of the picture, the reduction of "flicker" (the human observer's residual perception of the image refresh interval corresponding to frame-by-frame or field-by-field transmission), etc. Some proposed high-definition video standards would (in effect) encode the video information as two parallel serial streams, to preserve downward standards compatibility.
The electronic waveform also contains various timing signals transmitted between the horizontal lines and frames, such as horizontal line synchronization, frame synchronization, and the blanking interval, which are utilized to assure accurate reproduction of the video picture from the electronic waveform. This timing information is generally included on the waveform as pulses interspersed between various sections of modulated pixel information.
Many image processing techniques which have been proposed or are currently in use involve modifying the value of a pixel of a video image with reference to the values of neighboring pixels, and producing a new or better image based on this comparison. These techniques have been suggested for noise reduction generally, and for specific applications such as target or object identification, low light surveillance, or the improvement of hazy or unsharp images. Also, these techniques have been suggested in combination with known image or object recognition methods or techniques.
The Laplacian algorithm, for example, is employed to enhance the detail of an image. The application of this algorithm involves averaging the intensities of a set of pixels which surround a certain pixel of interest, and subtracting some fraction of the average value of intensity from the intensity of the pixel of interest. This difference is then substituted for the intensity of the pixel of interest.
Some description of current picture processing and filtering techniques is found in Topics in Applied Physics--Picture Processing and Digital Filtering, Volume 6 edited by T. S. Huang, which is incorporated herein by reference.
The comparison of pixel values can be done digitally. For example, the value of each pixel could be measured, and the value placed in a Random Access Memory (RAM) device. The pixel values could then be selectively recalled from the RAM device, to be compared with neighboring pixels and modified accordingly, using digital arithmetic operations in a digital processing device. With the great versatility of the currently available digital processing devices, a very wide range of algorithms may be implemented using these techniques.
Unfortunately, the speed of the digital processing devices in the current technology is insufficient to allow real-time video processing for many applications without considerable difficulty and/or cost. In many instances, it is desirable to be able to process or improve a video image in real time. Also, since the pixel information is usually in analog form, the digital implementation of these suggested algorithms will, of course, require additional hardware for analog to digital and digital to analog conversion at video rates.
It has been proposed that satisfactory results could be achieved by using analog techniques to process video images in real time, if a two-dimensional array of pixels could be produced from the video signal. These analog techniques would involve electrically combining spatially related pixel values (e.g. using resistor networks and/or op amps) to implement a desired transform operation on each pixel, and thus produce the new image.
Such analog techniques require the simultaneous availability of the pixel values which are to be combined. However, no random-access structure organization for storing analog signals at video data rates is readily available. Therefore, real-time comparison of horizontally or vertically related pixels in a raster-scanned video input signal requires the use of serial access memory devices or signal delay devices.
Signal delay devices produce an output signal which is a time-delayed version of the input signal. If a video signal is being input to a signal delay device, the spatial relationship between the pixel being input to the delay device and the pixel simultaneously being output by the delay device is determined by the amount of time delay introduced by the delay device.
The amounts of delay time commonly utilized in image processing circuitry are individual pixel delays, horizontal line delays, and field delays. Recalling the previously mentioned raster scanning method, for example, an individual pixel delay device will output the pixel to the immediate left of the input pixel; a horizontal line delay will output the pixel vertically above the input pixel; and the field delay device will output a pixel which has almost the same horizontal and vertical position as the current input pixel, but which is displaced from it by one line in an interlaced display. (In a non-interlaced display, the output pixel will correspond to the pixel component of the previous field which has the same (x,y) coordinates as the current pixel.)
It can be seen that the combination of the various types of delays may be used to simultaneously make available a desired set of pixel values with a desired spatial relationship for analog real-time processing. The set of pixels commonly may be a 3-by-3 or 5-by-5 grid, a horizontal or vertical rectangle, or generally may be any set of pixels one may envision.
In an iterated procedure, this set of pixels "travels" through the raster-scanned image. In the particular combination which produces a 3-by-3 grid of pixels, for example, this traveling grid begins in the upper left corner of the field and travels line by line until reaching the lower right corner of the field. Conventional high-speed operational amplifiers, for example, may then be used to weight and sum the pixel values produced by the delay devices.
These analog processing techniques have the potential to be realized in real time, subject to the limitation, of course, that it will not be possible to output a pixel from the image processing circuits until all pixels upon which the output pixel depends have been input.
A great deal of work has been invested in the art into defining various video transforms based on sequential manipulations of small blocks of pixels, e.g., 3 by 3 blocks of pixels. As recognized in the previous literature of this kind, not only can different-shaped blocks of pixels be used as the input to the sequentially iterated state of a transformed algorithm, but the blocks themselves do not have to be solid blocks of pixels. That is, one option is to use a sparse grid of pixels (e.g., a small checkerboard-type pattern) to provide the input values for the local transformation of each pixel value. Similarly, not all of the pixels whose values are input into the transformation algorithm need be weighted the same. One advantage of analog methods, such as those enabled by the present invention, is that analog weighting can be easily used, simply by specifying ratios between resistors (and/or between capacitors). Thus, while the principally preferred best mode (as extensively discussed below) uses the Laplacian transform algorithm, a tremendous variety of other transform algorithms could be used.
An example of a device which embodies several of the previously discussed concepts is disclosed in U.S. Pat. No. 4,399,461 to Powell. The Powell patent discloses the use of field delays, line delays, and pixel delays to produce a plurality of signals representative of pixels in the video image. Various video processing techniques may then be employed by multiplying and summing these signals. The Powell patent specifically discloses image enhancement by improving contrast in vertical, horizontal, or diagonal directions.
A general description of the various video processing algorithms which may be accomplished by a device such as is disclosed in the Powell patent may be found in an article by Benjamin M. Dawson entitled, "Technology Trends--Image Filtering for Edge Enhancement," published in Photonics Spectra, February, 1986. Discussed therein are video processing techniques such as edge enhancement, shift-and-difference, gradient, and Laplacian transform.
The use of signal delay devices to keep a running sum of the serially input pixel values has been suggested. U.S. Pat. No. 4,231,065 to Fitch et al. discloses the use of recursive filters to process a video signal. The algorithm employed therein causes a moving average of the intensity to be subtracted from the input signal to produce a new signal containing enhanced local variations of the original signal.
The use of a CCD shift register as a signal delay device in a video processing circuit is disclosed in U.S. Pat. No. 4,568,977 to Chamberlain et al. In the Chamberlain patent, the Laplacian image enhancement algorithm is performed on the input signal. Line delays and pixel delays are realized by a single, very long shift register. The multiplication and addition functions of the Laplacian algorithm are performed on the same chip which contains the CCD shift register.
In U.S. Pat. No. 4,096,516 to Pritchard, the chrominance and luminance components of a standard color television signal are separated using several one line delay devices. The delay time of the line delay devices is controlled by a clock signal. The clock signal is generated by a frequency multiplier and is based on the frequency of the incoming chrominance signal.